Crate datafusion_physical_plan

Traits for physical query plan, supporting parallel execution for partitioned relations.

Entrypoint of this crate is trait ExecutionPlan.

Re-exports

• pub use crate::display::DefaultDisplay;
• pub use crate::display::DisplayAs;
• pub use crate::display::DisplayFormatType;
• pub use crate::display::VerboseDisplay;
• pub use crate::execution_plan::collect;
• pub use crate::execution_plan::collect_partitioned;
• pub use crate::execution_plan::displayable;
• pub use crate::execution_plan::execute_input_stream;
• pub use crate::execution_plan::execute_stream;
• pub use crate::execution_plan::execute_stream_partitioned;
• pub use crate::execution_plan::get_plan_string;
• pub use crate::execution_plan::with_new_children_if_necessary;
• pub use crate::execution_plan::ExecutionMode;
• pub use crate::execution_plan::ExecutionPlan;
• pub use crate::execution_plan::ExecutionPlanProperties;
• pub use crate::execution_plan::PlanProperties;
• pub use crate::metrics::Metric;
• pub use crate::stream::EmptyRecordBatchStream;

Modules

• aggregates
  Aggregates functionalities
• analyze
  Defines the ANALYZE operator
• coalesce_batches
  CoalesceBatchesExec combines small batches into larger batches.
• coalesce_partitions
  Defines the merge plan for executing partitions in parallel and then merging the results into a single partition
• common
  Defines common code used in execution plans
• display
  Implementation of physical plan display. See crate::displayable for examples of how to format
• empty
  EmptyRelation with produce_one_row=false execution plan
• execution_plan
• explain
  Defines the EXPLAIN operator
• expressions
  Defines physical expressions that can evaluated at runtime during query execution
• filter
• functions
  Deprecated module. Add new feature in scalar_function.rs
• hash_utils
  Functionality used both on logical and physical plans
• insert
  Execution plan for writing data to DataSinks
• joins
  DataFusion Join implementations
• limit
  Defines the LIMIT plan
• memory
  Execution plan for reading in-memory batches of data
• metrics
  Metrics for recording information about execution
• placeholder_row
  EmptyRelation produce_one_row=true execution plan
• projection
  Defines the projection execution plan. A projection determines which columns or expressions are returned from a query. The SQL statement SELECT a, b, a+b FROM t1 is an example of a projection on table t1 where the expressions a, b, and a+b are the projection expressions. SELECT without FROM will only evaluate expressions.
• recursive_query
  Defines the recursive query plan
• repartition
  This file implements the RepartitionExec operator, which maps N input partitions to M output partitions based on a partitioning scheme, optionally maintaining the order of the input rows in the output.
• sorts
  Sort functionalities
• spill
  Defines the spilling functions
• stream
  Stream wrappers for physical operators
• streaming
  Generic plans for deferred execution: StreamingTableExec and PartitionStream
• tree_node
  This module provides common traits for visiting or rewriting tree nodes easily.
• udaf
• udf
• union
  The Union operator combines multiple inputs with the same schema
• unnest
  Define a plan for unnesting values in columns that contain a list type.
• values
  Values execution plan
• windows
  Physical expressions for window functions
• work_table
  Defines the work table query plan

Macros

• handle_state
  The handle_state macro is designed to process the result of a state-changing operation. It operates on a StatefulStreamResult by matching its variants and executing corresponding actions. This macro is used to streamline code that deals with state transitions, reducing boilerplate and improving readability.
• internal_err
  Macro wraps Err($ERR) to add backtrace feature

Structs

• ColumnStatistics
  Statistics for a column within a relation
• Statistics
  Statistics for a relation Fields are optional and can be inexact because the sources sometimes provide approximate estimates for performance reasons and the transformations output are not always predictable.
• TopK
  Global TopK

Enums

• ColumnarValue
  The result of evaluating an expression.
• Distribution
  How data is distributed amongst partitions. See Partitioning for more details.
• InputOrderMode
  Specifies how the input to an aggregation or window operator is ordered relative to their GROUP BY or PARTITION BY expressions.
• Partitioning
  Output partitioning supported by ExecutionPlans.

Traits

• Accumulator
  Tracks an aggregate function’s state.
• AggregateExpr
  An aggregate expression that:
• ExecutionPlanVisitor
  Trait that implements the Visitor pattern for a depth first walk of ExecutionPlan nodes. pre_visit is called before any children are visited, and then post_visit is called after all children have been visited.
• PhysicalExpr
  See create_physical_expr for examples of creating PhysicalExpr from Expr
• RecordBatchStream
  Trait for types that stream arrow::record_batch::RecordBatch
• WindowExpr
  Common trait for window function implementations

Functions

• accept
  Visit all children of this plan, according to the order defined on ExecutionPlanVisitor.
• project_schema
  Applies an optional projection to a SchemaRef, returning the projected schema
• visit_execution_plan
  Recursively calls pre_visit and post_visit for this node and all of its children, as described on ExecutionPlanVisitor

Type Aliases

• SendableRecordBatchStream
  Trait for a Stream of RecordBatches